Revolutionary Database Technology for Data Intensive Research

The heart of a scientific data warehouse is its database system, running on a modern distributed platform, and used for both direct interaction with data gathered from experimental devices and management of the derived knowledge using workflow software. However, most (commercial) DBMS offerings cannot fulfill the demanding needs of scientific data management. They fall short in one or more of the following areas: multi-paradigm data models (including support for arrays), transparent data ingestion from, and seamless integration of, scientific file repositories, complex event processing, and provenance. These topics only scratch the surface of the problem. The state of the art in scientific data exploration can be compared with our daily use of search engines. For a large part, search engines rely on guiding the user from their ill-phrased queries through successive refinement to the information of interest. Limited a priori knowledge is required. The sample answers returned provide guidance to drill down, chasing individual links, or to adjust the query terms. The situation in scientific databases is more cumbersome than searching for text, because they often contain complex observational data, eg telescope images of the sky, satellite images of the earth, time series or seismograms, and little a priori knowledge exists. The prime challenge is to find models that capture the essence of this data at both a macro-and micro-scale. The answer is in the database , but the 'Nobel-winning query' is still unknown. Next generation database management engines should provide a much richer repertoire and ease of use experience to cope with the deluge of observational data in a resource-limited setting. Good is good enough as an answer, provided the journey can be continued as long as the user remains interested. We envision seven directions of long term research in database technology: • Data Vaults. Scientific data is usually available in self-descriptive file formats as produced by advanced scientific instruments. The need to convert these formats into relational tables and to explicitly load all data into the DBMS forms a major hurdle for database supported scientific data analysis. Instead, we propose the data vault, a database-attached external file repository. The data vault creates a true symbiosis between a DBMS and existing file-based repositories, and thus provides transparent access to all data kept in the repository through the DBMS's (array-based) query language. • Array support. Scientific data management calls for DBMSs that integrate the genuine scientific data model, multi-dimensional arrays, …